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I. Introduction 

Data mining, the extraction of hidden predictive information from large databases , is a powerful new 
technology with great potential to help companies focus on the most important information in their data 
warehouses. Data mining tools [3] predict future trends and behaviors, allowing businesses to make proactive, 
knowledge-driven decisions. The automated, prospective analyses offered by data mining [8,9] move beyond 
the analyses of past events provided by retrospective tools typical of decision support systems. Data mining 
tools can answer business questions that traditionally were too time consuming to resolve. It is interactive and 
iterative, involving the following steps: 

• Step 1. Application domain identification: Investigate and understand the application domain and the 

relevant prior knowledge. In addition, identify the goal of the KDD from the administrators’ or users’ 
point of view. 

• Step 2. Target dataset selection: Select a suitable dataset, or focus on a subset of variables or data 

samples where data relevant to the analysis task are retrieved from the database. 

• Step 3. Data Preprocessing: the DM basic operations include ‘data clean’ and ‘data reduction’: In the 

‘data clean’ process, we remove the noise data, or respond to the missing data field. Inthe ‘data 
reduction’ process, we reduce the unnecessary dimensionality or adopt useful transformation methods. 
The primary objective is to improve the effective number of variables under consideration. 

• Step 4. Data mining: This is an essential process, where AI methods are applied in order to search for 

meaningful or desired patterns in a particular representational form, such as association rule mining, 
classification trees, and clustering techniques. 

• Step 5. Knowledge Extraction: Based on the above steps it is possible to visualize the extracted patterns 

or visualize the data depending on the extraction models. Besides, this process also checks for or 
resolves any potential conflicts with previously believed knowledge. 
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• Step 6. Knowledge Application: Here, we apply the found knowledge directly into the current 

application domain or in other fields for further action. 

• Step 7. Knowledge Evaluation: Here, we identify the most interesting patterns representing knowledge 

based data on some measure of interest. Moreover, it allows us to [2] improve the accuracy and 
efficiency of the mined knowledge. 



I 
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Fig.l. Data mining phases 



The revolution in database technologies has resulted in an increase of data accumulation in many 
areas, such as financial, marketing, sports and the biological and [29] medical sciences. Feature selection has 
many advantages such as shortening the number of measurements, reducing the execution time and improving 
transparency and compactness of the suggested diagnosis. 

1.1 Feature selection 

Feature selection is the process of selecting a subset of d’ features of the set D, such that d<D. The 
primary purpose of feature selection is to reduce the computational cost and to improve the performance of the 
learning algorithm. Feature selection algorithms [30, 31, 32] deal with different evaluation criteria and 
generally, are classier into filter and wrapper models[31]. The filter model evaluates the general characteristics 
of the training data to select a feature subset without relation to any learning algorithms; thus, it is 
computationally economical. 

1.2 Feature selection strategies 

This subsection explains three different criteria for the feature selection process. The features are 
regulated with respect to decreasing values of the fuzzy entropy. A feature in the first position is the most 
relevant and the one in the last position is the least relevant in the resulting rank vector. 

Mean Selection (MS) Strategy: A feature f E F is selected if it satisfies the following condition: 






ft * i*i 



Where ■?(/) is the relevance value of the features, which is selected if it is greater than or equivalent to the 
mean of the relevant values. This strategy will be useful in examining the suitability of the fuzzy entropy 
relevance measure. 



Half Selection (HS) Strategy: The half selection strategy aims to reduce feature dimensionality to select 
approximately 50% of the features in the domain. The feature / E F is selected if it satisfies the following 
condition: 

^ -v 
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Where is the position of the feature in the rank vector. It represents the selected features having a relevance 
value higher than a given threshold, which is calculated as IFI/2 . This strategy does produce great reductions, 
close to 50%. At the same time, some of the selected features are irrelevant despite them passing the threshold. 
Neural Network for Threshold Selection (NNTS): an ANN is one of the well-known machine learning 
techniques and it can be used in a variety of applications in data mining. The ANN provides a variety of feed 
forward networks that are generally called back propagation networks. The output value can be considered as a 
threshold value of the given fuzzy entropy. 

II. Literature Review 

Recently ,a number of researchers have focused on several feature selection methods and most of them 
have reported their good performance in database classification. Battiti [7] proposes a method called Mutual- 
information-based Feature Selection(MIFS),in which the selection criterion is based on maximizing the mutual 
information between candidate features and the class variables and minimizing the redundancy between 
candidate features and the selected features. Hanchuan et al. [8] follow a similar technique to MIFS , which has 
been called the minimal-redundancy-maximal-relevance(mRMR)criterion. 

It eliminates them annually tuned parameter with cardinality of the features already selected. 
Pabloetal. [9] present a Normalized Mutual Information Feature Selection algorithm. YuandLiu [10] developed 
a correction-based method for relevance and redundancy analysis and then removed redundant features using 
the Markov Blanket method. In addition features election methods are analyzed by a number of techniques. 
Abdel- Aal [1] developed a novel technique for feature ranking and selection with the group method of data 
handling. Feature reduction of more than 50% could be achieved and improved in the classification 
performance. Sahanetal. [11] built a new hybrid machine learning method for a fuzzy-artificial immune system 
with ak-nearest neighbor algorithm to solve medical diagnosis problems, which demonstrated good results. 
Jaganathanetal. [12] applied a new improved quick reduct algorithm, which is a variant of quick reduct for 
feature selection and test editon a classification algorithm called Ant Miner. Sivagami- nathanetal. [13] 
proposed a hybrid method combining Ant Colony Optimization and Artificial Neural Networks(ANNs)to deal 
with feature selection, which produced promising results. Linetal. [14] proposed Simulated Annealing 
approach for parameter setting in Support Vector Machines, which is compared with a grid search parameter 
setting and was found to produce higher classification accuracy. Lin etal. [15] applied a Particle -Swarm- 
Optimization-based approach to search for appropriate parameter values for a back-propagation network to 
select the most valuable subset of features to improve classification accuracy. Unleretal. [16] developed a 
modified discrete particle swarm optimization algorithm for the feature selection problem and compared it 
with tabuand scatter search algorithms to demonstrate it effectiveness. Changetal. [17] introduced a hybrid 
model for integrating a case-based reasoning approach with a particle swarm optimization model for feature 
subset selection in medical database classification. Salamoetal. [18] evaluated a number of measures for 
estimating feature elevance based on rough set theory and also proposed three strategies for feature selection in 
a Case Based Reasoning classifier. Qasemetal. [19] applied a time variant multi- objective particle swarm 
optimization to an RBF Network for diagnosing medical diseases. The use of discriminant analysis is ever 
increasing (Wilson and Sharda, 1994; Altman, Haldeman, and Narayanan, 1977). The Altman Z-score model 
is predominately used in discriminant analysis (Jo, Han and Lee, 1997). 



Work 


Classifier 


No.of datasets 


R.E.Abdel-Aal(2005) 


GMDH 


6 


M.F.Akay(2009) 


SVM 


2 


Chin-Yuan Fan(2011) 


HM 


1 


Huan Liu(2005) 


IFSA 


4 


R.Battiti(1994) 


NN 


4 


Pablo A.Extevez(2009) 


NN 


2 


Serai Sahan(2007) 


K-nn 


3 


P.Jagaathan(2007) 


ACO 


1 


Rahul Karthik Sivagaminathar 
(2007) 


ACO 


1 


Shih-Wei Lin(2008) 


SVM 


4 
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Zne-Jung Lee(2008) 


SVM 


4 


Tsung-Yuan Tseng(2008) 


SVM 


4 


Alper Unler(2010) 


Swarm optimization 


3 


Pei-Chann Chang(2012) 


Swarm optimization 


3 


Maria Salamo(2011) 


Feature selection 


2 


Sultan Noman Qasem(2011) 


RBF 


1 


H.M.Lee(2001) 


Feature selection 


3 


S.Rajasekaran(2010) 


NN 




Resul Das(2009) 


NN 


4 


Hanchuan Peng (2005) 


Feature selection based on 
mutual information 


8 


Chin- Yuan Fan(2011) 


hybrid model 


1 


Yang C, Sudderth J(2009) 


Akt signaling 


69 


Y. Saeys, I. Inza,(2007) 


Feature selection 


19 


Parpinelli, R(2002) 


ant colony optimization 


4 


Ooghe, H. and Spaenjers(2010) 


BFPM 


1 


Fuellerer, G(2009) 


Ant colony optimization 


3 


Jia Yu, Yun Chen(2011) 


ant colony optimisation 


35 


Gao X, Bu D(2009) 


BMC Structural Biology 


1 


Stoothoff W(2009) 


Neurochem 


11 


Brown G,(2012) 


Feature selection 


27 


Dramiski M,(2008) 


Feature selection 


110 


Sun X, Liu Y, Li(2012) 


Feature evalution 


45 



SVM= Support Vector Machine, DT= Decision Tree,LR=Logistic Regression, NN=Neural Networks, UCI 
repository machine, ACO=Ant Colony Optimization, RBF=Radial Bias Function, GMDH=Group Method of 
Data Handling based future ranking &selection method, IFSA=Integrating Feature Selection 
Algorthim,HM=Hybrid Model, BFPM=Business Focused Project Model. 



III. Background Details 

3.1 Fuzzy Entropy Relevant Measure 

In information theory, the Shannon entropy measure is generally used to characterize the impurity of 
a collection of samples. Assuming X as a discrete random variable with a finite set of n elements, where 
X={xl, x2, x3,...,xn}, then if an element xi occurs with probability p(%d [20], the entropy H(X) of X is 
defined as follows: 



1=0 



where n denotes the number of elements. 

An extension of Shannon entropy with fuzzy sets, which is to support the evaluation of entropies, is 
called fuzzy entropy. The proposed fuzzy entropy method is based on the utilization of the Fuzzy C -Means 
Clustering algorithm (FCM), which is used to construct the membership function of all features. The data may 
belong to two or more clusters simultaneously and the belonging of a data point to the clusters is governed by 
the membership values [24]. The FCM algorithm is explained as follows [25]. 

Step 1: assume the number of clusters (C), where 2 < c < A r , C - number of clusters and N - number of data 
points 

Step 2: calculate the jth cluster centerC-- using the following expression 






c i = ' 



MTj -U 






where g>l is the fuzziness coefficient and pij is the degree of membership for the ith data point X- L in cluster j. 
Step 3: calculate the Euclidean distance between the ith data point and the jth cluster center as follows: 



d jj — ICj-^rl 

Step 4: update the fuzzy membership values according to d { . ; . If d{ > 0, then 
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M = 



1 




d=0, then the data point coincides with the jth cluster center (C) and it will have the full membership value, 

i.e.,fiy =1.0 

Step 5: repeat Steps 2-4 until the changes in [a] are less than some pre-specified values. 

The FCM algorithm computes the membership of each sample in all clusters and then normalizes it. This 
procedure is applied for each feature. The summation of membership of feature ‘x’ in class ‘c’, divided by the 
membership of feature ‘x’ in all ‘C’ classes, is termed the class degree CI?. : (^)[2i5], which is given as 

CD c (A)= 




wherea^ denotes the membership function of the fuzzy set and Pa denotes the membership grade of x 
belonging to the fuzzy set A. 

The fuzzy entropy FE C . (JQ of class ‘c’ is defined as 

The fuzzy entropy FE(A') of a fuzzy set X is defined as follows: 

fe(A) = 

The probability p(X[) of Shannon's entropy is measured by the number of occurring elements. In contrast, the 
class degree CD c (df) in fuzzy entropy is measured by the membership values of the occurring elements. 
Suppose the elements are divided into a number of intervals: / 1? / 2 , then the Shannon entropy of interval is 
equal to the interval / 2 . However, fuzzy entropy can indicate that interval /i is distinguishable from interval / 2 . 
The difference between Shannon's entropy and the proposed fuzzy entropy is illustrated in Fig. 1, where the 
symbols ‘M’ and T’ denote male and female samples, respectively. The computation of Shannon's P. 
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Fig 1. Distribution of samples 
entropy in the intervals ^ and is as follows. 

H(/ l ) = -CpCM) iog 2 p(,\i) + p(F) log 2 p(F')'] 




11^3= -gfaft J + 9^0.72 

H(/ 2 > + —0.72 



Based on Eq. (5), the class degrees of ‘M’ and ‘F’ are 

, . 3.75 

CD m (A) = —— — — — = 0.033 
} 3,75-1-0,75 

0.75 

CD f (A) = — — — = 0,167 
^ ^ J 3,75 -I- 0,75 

The fuzzy entropy FE(jI) of a fuzzy set A is calculated as follows: 

FE(A) =fe m (A)+feM 
FE(A) = - {cD M {A)log 2 CD H {A) + CD F {A)log 2 

FE(i) = - (0.83310* 0.S33 + 0.167^0.167) = 0.651 
Similarly, the fuzzy entropy of the fuzzy set S is as follows: 
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^ = 1 4- 0.25 = 1.25 

X=M 

GO =1 + 1 + 1 + 0.75 = 3.25 

The class degree of ‘M’ is 1. 25/(1. 25+3.75)=0.25 and 3.75/ (1.25+3.75)=0.75. 

The fuzzy entropy FE(5) is 

FE(fi) = -C0.2Sfa fc 0.2S + 0.7Slo fc 0.7S) = 0.S11 
From the above results, Shannon's entropy of interval is equal to interval / 2 . Nonetheless, the fuzzy 
entropy indicates that interval is distinct from interval / 2 . The highest fuzzy entropy value of the feature is 
regarded as the most informative one. 

IV. Workflow Of Ant-Miner Algorithm 

The flowchart of the ant-miner algorithm is given below. The algorithm starts with obtaining a 
training set which consists of training cases. After that, the main loop will be executed to discover one rule per 
iteration: (1) It begins with initializing the index of [11,12,13,14] ant (t), the index of converge, 
Convergelndex (j) which is used to test convergence of ants paths and pheromone on all trails. Convergence of 
path is an important indicator to check if a steady path chosen by ant colony has formed. It tests if ants take the 
same path one after another and record how many ants take this path. (2) A sub-loop is executed to discover 
classification rules by a number of artificial ants (No_of_ants) who explore paths in turn. And the discovery 
process consists of three main steps: rule generation, rule pruning and pheromone updating. 

The sub-loop will terminate under the condition that all ants have taken their exploration 
(t>No_of_ants), or the convergence state has been reached (j>No_rule_converge). No rule converge is the 
threshold of Converglndex (j). It means if there are No_rule_con verge ants take the same path one after 
another, the path is qualified for a discovered rule candidate. If the current ant has constructed a rule that is 
exactly the same as the rule constructed by previous ants, then it is said the ants have converged to a single rule 
(path) and the value of Convergelndex (j) will increase by 1. (3) The main loop selects the best rule from the 
discovered rules according to their qualities. (4) The training cases which are covered by the best rule need to 
be removed from the training set. In other words, the number of training cases in the training set is gradually 
decreased with continuous matching with the best rules. 

4.1 Datasets Description 

The proposed system is evaluated by eight benchmarks datasets: diabetes, tic-tac-toe, chess, car, 
hepatitis, fiber, liver disorder, heart statalog which is taken from various fields life, game, industry of UCI 
Machine learning repository. 



Dataset 


Area 


No. of 
instances 


No. of features 


No. of classes 


Diabetes 


Life 


768 


8 


2 


Tic-tac-toe 


Game 


958 


9 


2 


Chess 


Game 


42 


6 


2 


Car 


industry 


1728 


6 


2 


Hepatitis 


Life 


155 


19 


2 


Fiber 


Industry 


48 


4 


2 


Liver disorder 


Life 


345 


7 


2 


Heart statlog 


Life 


270 


13 


2 
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END 



Fig.4. Work flow of Ant-Miner Algorithm for classification rule discovery 
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V. Results and Discussions 

The classification performance of the proposed feature selection method is measured by using an RBF 
Network classifier, bagging, AdaBoostMl and stacking. An RBF network is a type of ANN, which is a simple 
network structure with better approximation capabilities. 

5.1 Pima Indians Diabetes 

It contains 768 instances described by eight features used to predict the presence or absence of 
diabetes. The features are as follows: (1) number of pregnancies, (2) plasma glucose concentration, (3) 
diastolic blood pressure, (4) triceps skin fold thickness, (5) serum insulin, (6)body mass index, (7)diabetes 
pedigree function and (8)age in years. It is found that Ant-Miner yields the highest accuracy of 99.73 when 
the[3] Neural Network for Threshold Selection strategy employed which is shown in table result 1 . 

5.2 Hepatitis 

There are 19 features (age, sex, steroid, antivirals, fatigue, malaise, anorexia, liver big, liver film, 
spleen palpable, spiders, ascites, varices, bilirubin, alk phosphate, SGOT, albumin, protime and histology). It 
is found that Ant-Miner yields the highest accuracy of 98.70 when Neural Network for Threshold Selection 
strategy was performed which is shown in table result 1. 

5.3 Heart-Statlog 

It is described by 13 features (age, sex, chest, resting blood pressure, serum cholesterol, fasting blood 
sugar, resting electro cardio graphic, maximum heart rate, exercise induced angina, old peak, slope, number of 
major vessels and thal). It is found that Ant-Miner yields the highest accuracy of 99.25 when the Neural 
Network for Threshold Selection strategy was employed which is shown in table result 1. 

5.4 Liver-disorder 

This dataset contains mcz, alkphos, sgpt, sgot, gammagt, drinks as 6 attributes for 187 instances with 
two classes 1 or 2. It is found that Ant-Miner yields the highest accuracy of 98.93 when Neural Network for 
Threshold Selection strategy was performed in table result 1. 

5.5 Tic-tac-toe 

This dataset contains 51 instances with 9 attributes like top -left- square, top-middle-square, middle- 
left-square, middle-middle- square, middle-right- square, bottom-left- square, bottom-middle- square and bottom- 
right-square with two classes. It is found that Ant-Miner yields the highest accuracy of 97.41 in Neural 
Network for Threshold Selection strategy which is shown in table result 1. 

5.6 Chess 

This dataset consist of 6 attributes namely l)White_king_file, (2)White_king_rank 
(3)White_rook_file, (4) White_rook_rank, (5) Black_king_file ,(6) Black_king_rank and two classes like win 
or lose for 42 instances. It is found that Ant-Miner yields the highest accuracy of 97.61 in Neural Network for 
Threshold Selection strategy which is shown in table result2. 

5.7 Car 

This contains 1117 instances with 6 attributes like buying, maint, doors, persons, lug_boot and safety 
with two classes. It is found that Ant-Miner yields the highest accuracy of 97.61 in Neural Network for 
Threshold Selection strategy which is shown in table result2. 

5.8 Fiber 

The classification result for the Car dataset is shown in Table8. The classified sensitivity of 94.11, 
specificity of 96.77 is arhieved with the Neural Network for Threshold Selection strategy in Ant-Miner. This 
dataset contains two classes yes or no with 4 attributes. The attributes considered are cracker, diet, and subject 
and digested. It is found that Ant-Miner yields the highest accuracy of 97.91 in Neural Network for Threshold 
Selection strategy shown in table result2. 

The result obtained after processing with several strategies and classification methodologies the 
Accuracy can be shown in table 
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Table result 1 



SI. 

30 


Ezlzsei 

Nzme 


Slrzlezy 


Select 

Feartmues 


FHF 

C03fisi03 


SlzcJcdnz 

confjjion 


Bz^inz 
ca infusion. 


AdzBooilMl 

confjsioa 


Art -Minor 
confjjiozi 


1 


DizbeTse 


None 


All 


7 5 39 


-55 10 


75.39 


7434 


9930 


ms 


23 


93.30 


■55.10 


99 


993-6 


99 47 


US 


1-4-5 jS- 
13 


9^09 


do. 10 


93 


993d 


93 17 


KN TS 


23 


93.30 


do. 10 


99 


99 2d 


99.73 


2 


Hepzlilis 


Kane 


All 


35 30 


7935 


31.29 


3233 


933d 


MS 


1.15. Id. 
IS 


94.d5 


57.21 


P7.42 


9d.7Q 


9 7.41 


BS 


23A3. 

633$. 

10-11-12. 

13.14.17. 

19 


94.-55 


|ji 

“j 

ki 


9732 


9639 


9-5 12 


NNTS 


1.15. Id. 
13 


3431 


7 9 35 


3937 


39.03 


93.70 


3 


HezH 
S-lz.1 ]oj 


None 


AH 


34.07 


55.55 


31 11 


30 


9333 


MS 


1.43,3 


9-531 


5535 


9 7 .d2 


9732 


97.77 


BS 


233.7. 

ejo. 

11.12.13 


93.51 


55.55 


9932 


9932 


97.03 


MKTS 


1.433 


93.51 


55.55 


9932 


9932 


99 25 


4. 


Liver 

Disorder 


None 


AH 


■S3 10 


57.21 


70 33 


6~? 91 


9732 


MS 


U 


94-55 


57.21 


97.90 


9d.7Q 


97.5d 


BS 


3.433 


94-55 


57.21 


93 J1 


96 79 


9339 


NNTS 


123.4 


95.72 


57.21 


9^32 


96 79 


93JP3 


5. 


Tic-Isfir 

Toe 


None 


AH 


90.07 


od 3d 


90 11 


39 41 


90.19 


MS 


2.43.7 


91.07 


5636 


9230 


92.03 


92 15 


BS 


1333. 

9 


33.23 


5636 


9432 


93.03 


94 11 


NKTS 


All 


3431 


■935 


393 7 


39.03 


97.41 



Table result 2 



SI 


Dataset 


S-tratey 


Selected 


REF 


Stacking 


Bagging 


AdaBoostMl 


Ant-Miner 


no 


Name 




Features. 


contusion 


contusion 


contusion 


contusion 


contusion 






None 


All 


95.52 


71.42 


93.15 


92.60 


92. S5 


6. 


Ches-s. 


MS 


1,46 


SS.09 


71.42 


94.75 


95.85 


95.23 






HS 


2^,S 


SS.09 


71.42 


95.75 


94. S5 


95.23 






NNTS 


4,6 


SS.09 


71.42 


96.52 


96.23 


97.61 






None 


All 


90.06 


6S.66 


98.90 


95.16 


9S.S3 


7. 


Car 


MS 


3,4 


98.95 


6S.66 


96.50 


96.50 


9S.92 






HS 


1, 2,5,6 


94. S9 


6S.66 


97.96 


96.50 


97.94 






NNTS 


All 


SS.09 


71.42 


96.92 


96.92 


97.61 






None 


All 


56.25 


64.58 


56.25 


64 5 S 


95.75 


S. 


Fiber 


MS 


4 


91.66 


64.5S 


95.91 


94.91 


95. S3 






HS 


U3 


91.66 


64. 5S 


92.91 


95.91 


93.75 






NNTS 


3,4 


91.66 


64.58 


95.91 


95.91 


97.91 



Fig.4. Work flow of Ant-Miner Algorithm for classification rule discovery (Jia Yu et al., (201 1) 



I IJMER I ISSN: 2249-6645 I 



www.ijmer.com 



I Vol. 5 I Iss.3l Mar. 2015 I 401 



A Threshold fuzzy entropy based feature selection method applied 




Diabetes 




Hepatitis 




Heart-stadog 





Tk-tae-foe 
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450 




Car 

Where, 

NNTS KS MS None 



450 

400 

aso 

300 

250 

200 

ISO 

100 

SO 

0 




Fiber 




This paper demonstrated the Ant-miner based data mining approach to discover decision rules from 
experts’ decision process. This study is the first work on Ant-miner for the purpose of discovering experts. Four 
data mining techniques which are Radial Basis Function (RBF) network, Stacking, Bagging, AdaBoostMl and 
Ant-miner classification methodologies, are applied to compare their performance with that of the ant-miner 
method. The results of NNTS shows the better accuracy between experts and data-mining techniques ANT- 
MINER method is perfectly matched and significantly better than RBF network, Stacking, Bagging and 
AdaBoostMl methods. 



VI. Conclusion 

Data mining has been widely applied to various bench mark datasets. This work proposes a new 
method of classification rule discovery for accuracy prediction by using ant-miner algorithm.In performance 
terms, the techniques obtain different results depending on the performance factors chosen. Some techniques 
generate less rules and shows poor accuracy, but Ant-miner provides more rules and gives better predictive 
accuracy when compare to other techniques. This study has conducted a case study using public dataset for 
analysis and benchmark approval from UCI repository. Finally this paper suggests that Ant-miner could be a 
more suitable method than the other classifiers like Radial Basis Function (RBF) 
network, Stacking, Bagging, AdaBoostMl and Ant-Miner. In future research, additional artificial techniques 
could also be applied. And certainly researchers could expand the system with more dataset. 
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